Papers with uniform sampling
Resisting the Lure of the Skyline: Grounding Practices in Active Learning for Morphological Inflection (2024.acl-short)
Copied to clipboard
| Challenge: | Several approaches to active learning are available, including confidence-based, diversity-based and committee-based. |
| Approach: | They propose to use a baseline and a skyline to measure the accuracy of the unannotated sample pool. |
| Outcome: | The proposed model outperforms a random selection baseline and a skyline approach. |
Anchor Points: Benchmarking Models with Much Fewer Examples (2024.eacl-long)
Copied to clipboard
| Challenge: | Modern language models exhibit powerful but brittle behavior, leading to larger and more diverse benchmarks. |
| Approach: | They propose to use anchor points to select small subsets of a language model-prompt dataset to capture model behavior across the entire dataset. |
| Outcome: | The proposed technique outperforms standard benchmarks in language models with 1-30 anchor points . the proposed technique can be used to compare models on different regions of the dataset . |
Learning Task Sampling Policy for Multitask Learning (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to train multi-task models with auxiliary tasks are limited by the number of combinations and the importance of each auxiliary task is not known a priori. |
| Approach: | They propose a search method that automatically assigns importance weights to auxiliary tasks to improve the target task quality. |
| Outcome: | The proposed method outperforms uniform sampling and the corresponding single-task baseline on XNLI and GLUE. |
HERMES: KV Cache as Hierarchical Memory for Efficient Streaming Video Understanding (2026.acl-long)
Copied to clipboard
| Challenge: | Existing models struggle to maintain stable understanding performance and low GPU memory overhead. |
| Approach: | They propose a training-free architecture for real-time and accurate understanding of video streams . HERMES reuses a compact KV cache, enabling efficient streaming understanding . |
| Outcome: | The proposed architecture achieves 10 faster TTFT compared to prior SOTA. |
Likelihood Variance as Text Importance for Resampling Texts to Map Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a language model map requires large text sets to be constructed . a resampling method reduces the number of texts needed while preserving accuracy of KL divergence estimates. |
| Approach: | They propose a method that selects important texts with weights proportional to log-likelihoods across models for each text. |
| Outcome: | The proposed method reduces the number of required texts while preserving the accuracy of KL divergence estimates. |
SampleMix: A Sample-wise Pre-training Data Mixing Strategy by Coordinating Data Quality and Diversity (2025.findings-emnlp)
Copied to clipboard
Xiangyu Xi, Deyang Kong, Jian Yang, Jiawei Yang, Zhengyu Chen, Wei Wang, Jingang Wang, Xunliang Cai, Shikun Zhang, Wei Ye
| Challenge: | Existing methods for pretraining data mixing for large language models neglect significant inter-domain overlaps and commonalities, failing to control the global diversity of the constructed training dataset. |
| Approach: | They propose a sample-wise data mixture approach that performs global cross-domain sampling by systematically evaluating the quality and diversity of each sample. |
| Outcome: | The proposed method exceeds existing domain-based methods in multiple downstream tasks and perplexity assessments. |
CritiQ: Mining Data Quality Criteria from Human Preferences (2025.acl-long)
Copied to clipboard
Honglin Guo, Kai Lv, Qipeng Guo, Tianyi Liang, Zhiheng Xi, Demin Song, Qiuyinzhe Zhang, Yu Sun, Kai Chen, Xipeng Qiu, Tao Gui
| Challenge: | Existing methods to train language models rely on manual design, perplexity, or careful prompt engineering. |
| Approach: | They propose a method that automatically mines criteria from human preferences for data quality with only 30 human-annotated pairs and performs efficient data selection. |
| Outcome: | The proposed method improves on human-annotated test sets and shows high accuracy on code, math, and logic domains. |
One QuantLLM for ALL: Fine-tuning Quantized LLMs Once for Efficient Deployments (2025.acl-long)
Copied to clipboard
| Challenge: | Quantization has shown promise for Large Language Models, but current methods require lengthy training to alleviate quantization loss. |
| Approach: | They propose to decouple weights and incorporate Low-Rank adapters to reduce weight sharing . they validate the approach on LLaMA2 families and Mistral on downstream evaluation . |
| Outcome: | The proposed approach shows high performance while reducing deployment time faced with multiple scenarios. |
ScaleBiO: Scalable Bilevel Optimization for LLM Data Reweighting (2025.acl-long)
Copied to clipboard
Rui Pan, Dylan Zhang, Hanning Zhang, Xingyuan Pan, Minrui Xu, Jipeng Zhang, Renjie Pi, Xiaoyu Wang, Tong Zhang
| Challenge: | Existing paradigms for bilevel optimization require second-order information, making it difficult to scale them up. |
| Approach: | They propose a scalable instantiation of a bilevel optimization paradigm for large-scale LLMs by using a memory-efficient training technique. |
| Outcome: | The proposed paradigm scales to 30B-sized LLMs on 8H100 GPUs. |
Group-Aware Reinforcement Learning for Output Diversity in Large Language Models (2025.emnlp-main)
Copied to clipboard
Oron Anschel, Alon Shoshan, Adam Botach, Shunit Haviv Hakimi, Asaf Gendler, Emanuel Ben Baruch, Nadav Bhonker, Igor Kviatkovsky, Manoj Aggarwal, Gerard Medioni
| Challenge: | Large Language Models suffer from mode collapse, repeatedly generating the same few completions even when many valid answers exist. |
| Approach: | They propose a group-aware policy optimization extension of GRPO that computes rewards over the group as a whole. |
| Outcome: | The proposed model improves on standard LLM benchmarks without compromising accuracy. |